Skip to content

beholder: add per-domain chip ingress emitters (ForDomain, EmitEvent) - #2504

Open
4of9 wants to merge 16 commits into
mainfrom
INFOPLAT-19544-beholder-domain-emitters
Open

4of9 wants to merge 16 commits into
mainfrom
INFOPLAT-19544-beholder-domain-emitters

Conversation

@4of9

@4of9 4of9 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

INFOPLAT-19544

What

  • beholder.Domain constants (data-feeds, data-streams, ccip) and Config.ChipDomainEmittersEnabled (off by default)
  • When enabled, NewGRPCClient builds one chip ingress emitter per domain:
    • own connection + batch buffer
    • sends the chainlink-domain: <domain> header
    • publishes with PublishBatchNoAck; NOP lookup on; client name beholder_<domain>
    • chip ingress only (never wrapped in the dual OTel emitter)
    • started and closed with the client
    • requires ChipIngressLogger; rejected by the HTTP client
  • Client.ForDomain(Domain) (DomainEmitter, error): lookup only; errors wrap ErrDomainEmittersDisabled or ErrUnknownDomain
  • DomainEmitter = Emitter + EmitEvent(ctx, ce.Event, callback): queues a fully built CloudEvent as given (also added to ChipIngressBatchEmitterService); the contract (cardinality, callbacks, no-ack semantics) is documented on the interface
  • Domain emitters reject events whose source is not their domain or a subdomain of it (data-feeds, data-feeds.*), so an event can't take another domain's route
  • Failure logs in the batch emitter (default and domain emitters) are rate-limited: the first failure per error code is logged, then at most one per minute with a suppressed_since_last_log count; chip_ingress.events_dropped still counts every event
  • Queueing-failure callbacks now run after the emitter's lock is released, so a callback that re-enters the emitter can't deadlock with Close (also fixes EmitWithCallback)
  • NewGRPCClient closes its chip ingress client if construction fails after creating it
  • Bump pkg/chipingress to 8ed7b172 (PublishBatchNoAck client side #2478) for batch.WithNoAck

Why

  • Node OTI telemetry is moving onto beholder, with each product domain served by its own chip ingress deployment. The load balancer routes on chainlink-domain (proven in the INFOPLAT-19544 spike).
  • Keeps chip ingress behind the beholder client.
  • Domains live in code because each one needs infra: a load balancer rule and a deployment.

Notes

  • Breaking for chipingress.Client implementers. The pin bump adds PublishBatchNoAck to the interface (via pb.ChipIngressClient). api-diff covers only the root module, so it doesn't flag this. Hand-written fakes must add the method when bumping, e.g. chainlink's noopChipIngressPublisher in core/services/synchronization/chip_ingress_batch_worker_test.go.
  • Behaviour changes for existing beholder users (tests assert the default emitter still uses acked PublishBatch with no routing header):

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ API Diff Results - github.com/smartcontractkit/chainlink-common

✅ Compatible Changes (12)

pkg/beholder (8)
  • Domain — ➕ Added

  • DomainCCIP — ➕ Added

  • DomainDataFeeds — ➕ Added

  • DomainDataStreams — ➕ Added

  • DomainEmitter — ➕ Added

  • DomainHeaderKey — ➕ Added

  • ErrDomainEmittersDisabled — ➕ Added

  • ErrUnknownDomain — ➕ Added

pkg/beholder.(*ChipIngressBatchEmitterService) (1)
  • EmitEvent — ➕ Added
pkg/beholder.(*Client) (1)
  • ForDomain — ➕ Added
pkg/beholder.Config (1)
  • ChipDomainEmittersEnabled — ➕ Added
pkg/beholder.writerClientConfig (1)
  • ChipDomainEmittersEnabled — ➕ Added

📄 View full apidiff report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pkcll pkcll left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a clean, additive change: default-off, reuses the batch emitter rather than forking it, and the in-process gRPC server tests (method, routing header, auth, resource attrs, NOP header, caller extensions) are a nice pattern. The cross-domain source guard is a good touch.

I ran the package locally at c2f5a8a (go test ./pkg/beholder/ -race: pass, 81.8% coverage; new tests ×10 with -race: no flakes) plus a few lifecycle/outage probes. Two things I'd like addressed, the rest is optional.

1. Domain emitters close their gRPC connection twice → Client.Close() always returns errors (blocking)

ChipIngressBatchEmitterService.stop() calls e.batchClient.Stop() and then e.ownedClient.Close(), but batch.Client.Stop() already closes the chipingress client (pkg/chipingress/batch/client.go, end of Stop). With the flag on, a clean start → Close() returns three extra errors:

rpc error: code = Canceled desc = grpc: the client connection is closing   (×3, one per domain)

It's hidden in the tests because every Close() result is discarded (t.Cleanup(func() { _ = client.Close() }), _ = client.Close() in the drain test).

Suggestion: drop the ownedClient.Close() from stop() (Stop owns the connection once started) but keep closing it on the construction-failure paths in newDomainEmitters / NewGRPCClient, and assert in a test that a start → close with domain emitters returns no domain-related error.

2. During an outage, every dropped event logs at Error (before enabling the flag on a fleet)

queue() logs failed to emit to chip ingress inside the per-message callback for every non-PublishError failure, and failed to queue message for chip ingress for every ErrMessageBufferFull. When one domain's chip ingress (or its LB rule) is unavailable — or the server doesn't have PublishBatchNoAck yet (Unimplemented, no fallback) — each node emits one Error line per telemetry event for that domain. This is pre-existing behaviour moved into queue(), but domain emitters are going to carry node telemetry volume, so the blast radius is new.

Could we rate-limit these (e.g. first failure + periodic summary per error_code), or log per batch and rely on chip_ingress.events_dropped{error_code} for per-event accounting, as the code already does for PublishError? Fine to do as a fast follow if you prefer, as long as it lands before the flag is turned on anywhere.

Tests worth adding

  • Outage/failure injection for domain emitters: server down, hung server + Close (callbacks fire exactly once), buffer full, Unimplemented.
  • Concurrent EmitEvent + Client.Close() under -race.
  • Construction-failure branches (newDomainEmitters is at 76%: NewClient error, batch-service ctor error, both cleanup defers).
  • Emit (attribute path) happy path on a domain emitter — only the reject case is covered.

Nits (optional)

  • var domains = []Domain{...} is a mutable package var; a func or a fixed-size array avoids accidental mutation.
  • append(append([]chipingress.Opt{}, opts...), ...) → slices.Concat(...).
  • svc.ownedClient / svc.domain are set after construction; passing them through the constructor keeps it in one place.
  • checkDomain is case-sensitive and requires the . separator; worth a line in the DomainEmitter doc.
  • Mention in the ChipIngressDomainEmittersEnabled doc that domain emitters are always batch emitters (independent of ChipIngressBatchEmitterEnabled), and the per-node cost: one buffer of ChipIngressBufferSize per domain; a connection only for domains actually used (dialing is lazy).

Questions

  1. What does the LB do with an unknown or missing chainlink-domain header (default backend vs reject, which gRPC status)?
  2. Is a chip-ingress server version with PublishBatchNoAck already deployed in stage/prod?
  3. Will consumers (OTI) need a beholdertest fake for DomainEmitter?

Out of scope here (I'll track separately)

  • pkg/chipingress/batch: splitMessagesByRequestSize recomputes proto.Size over the whole candidate batch for every appended message — O(n²), and ~90% of allocations on the send path in a memprofile. Not introduced by this PR, but this PR adds three more emitters on that path.
  • Client shutdown order: OTel providers are shut down before the batch/domain emitters drain, so drain-time metrics/spans are lost.

Happy to approve once (1) is fixed and there's an outage test.

@4of9

4of9 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @pkcll — great review. Fixes for (1), (2) and the tests are coming in follow-up commits. Answers to your questions:

1. Unknown/missing chainlink-domain: falls through to the default route (chip-ingress-public). Intended behaviour (still being settled): every deployment handles legacy-marked events, because topic and key come from the event's extensions, not the deployment. Misrouted telemetry still lands correctly. It shouldn't happen, though, so we'll add a metric, dashboard panel and alert for marked events served off their domain's deployment. Client-side, the source guard prevents cross-domain sends.

2. PublishBatchNoAck in stage/prod: not yet. atlas#13777 must be deployed to the domain deployments before the flag is turned on; there's no fallback (Unimplemented), which the new outage test covers.

3. beholdertest fake: not in this PR. We'll add one when the node work starts and we know what its tests need.

4of9 and others added 3 commits October 28, 2026 13:45
batch.Client.Stop already closes the chip ingress client, so stop() closing
it again made every Client.Close return one Canceled error per domain. The
connection is now only closed directly when construction fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers server down, hung server + Close (each callback fires once), buffer
full, a server without PublishBatchNoAck, concurrent EmitEvent and Close,
the Emit attribute path, and construction failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
During an outage the batch emitter logged one Error line per dropped event.
It now logs the first failure per error code, then at most one per minute
with a suppressed count. chip_ingress.events_dropped still counts every event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Domain list as a func so it cannot be mutated, slices.Concat for the domain
emitter options, and docs for the source check (case-sensitive, "."
separator) and for ChipIngressDomainEmittersEnabled (always batch emitters,
per-node cost).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@4of9

4of9 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks again @pkcll — all addressed:

1. Double close — 0d120a6. stop() no longer closes the connection (batch.Stop owns it); it's closed directly only if construction fails. Tests now assert Close() returns no client connection is closing errors, including the drain test.

2. Per-event error logs — 5770d39. First failure per error code is logged, then at most one per minute with a suppressed_since_last_log count; events_dropped still counts every event. Applies to the default emitter too, since they share queue().

Tests — 66af14e: server down, hung server + Close (each callback fires exactly once), buffer full, Unimplemented, concurrent EmitEvent + Close under -race, the Emit happy path, and construction failures. The batch-constructor failure branch isn't reachable via config (WithMaxGRPCRequestSize clamps), so it stays defensive. -race did catch one test-only race (a late callback logging via t after the buffer-full test ended), now fixed.

Nits — b3de7aa: allDomains() func, slices.Concat, domain passed through the constructor (in 0d120a6), and docs for case-sensitive . matching and always-batch / per-node cost.

Thanks for tracking the splitMessagesByRequestSize and shutdown-order items separately.

4of9 and others added 4 commits October 29, 2026 12:42
- Report queueing failures to the callback after the service lock is
  released, so a callback that re-enters the emitter cannot deadlock with
  Close (also fixes EmitWithCallback).
- Require ChipIngressLogger when domain emitters are enabled.
- ForDomain errors wrap ErrDomainEmittersDisabled or ErrUnknownDomain.
- Reject ChipIngressDomainEmittersEnabled in the HTTP client.
- Close the chip ingress client when NewGRPCClient fails after creating it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The invalid-event test now isolates CloudEvent validation (it passed on the
domain check before). New coverage: a cancelled context, domain emitters
with the batch emitter disabled, the per-domain client name, the logged
event source, and rate-limited buffer-full logs. The concurrency test no
longer calls require off the test goroutine and checks that emits overlap
Close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spell out what callers need: source and type must be low-cardinality,
EmitEvent sends the event as given (seqnum is overwritten, nothing is
stamped, the event must not be modified afterwards), when callbacks fire and
what a nil no-ack result means, that the source check is a client-side guard,
and that the Client must be started before Close to release connections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rsEnabled

Chip ingress is a background service users should not need to know about.
Existing ChipIngress* settings are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 9b4d2d3 Previous: f49aea8 Ratio
BenchmarkKeystore_Sign/nop/in-process 836.5 ns/op 225.1 ns/op 3.72
BenchmarkKeystore_Sign/hex/in-process 789.1 ns/op 287.9 ns/op 2.74

This comment was automatically generated by workflow using github-action-benchmark.

Comment thread pkg/beholder/client.go
if batchSvc != nil {
subs = append(subs, batchSvc)
}
for _, domain := range allDomains() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Drain domain emitters before shutting down OTel providers

The service engine runs the parent Close hook before closing sub-services. The parent hook calls c.close(), which shuts down the OTel meter provider; the domain batch emitters are drained afterwards. If a queued event completes during that drain, its callback records the sent/error/drop metric after the provider has shut down, so that outcome may not be exported.

Could teardown drain the domain emitters before shutting down the providers?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks — fixed for the domain emitters in 9b4d2d3. Client.close now drains them (concurrently) before the providers shut down, and a close-once wrapper makes the engine's later sub-service Close a no-op. TestForDomain_DrainsBeforeProvidersShutDown checks that the emitter has already drained and stopped when the provider shutdown runs.

I left the default batch emitter's ordering as it is to keep this PR scoped to the new code: it has drained after provider shutdown since 4a2d236, and Pavel flagged it as a separate follow-up. Happy to apply the same change there in its own PR.

The service engine runs the Client's Close hook, which shuts the OTel
providers down, before closing sub-services, so outcomes recorded while
the domain emitters drained were not exported. The hook now drains the
domain emitters first; a close-once wrapper makes the engine's later
sub-service Close a no-op. The default batch emitter is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@4of9

4of9 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@pkcll @thomaska ready for another look — the PR is at 9b4d2d3.

Since the last round:

  • Shutdown ordering (Thomas's P2): the domain emitters now drain before the OTel providers shut down (9b4d2d3). The default batch emitter keeps its existing order; that stays a separate follow-up.
  • Hardening from an internal review (79eb8d9, 6ca644e, 7ff0b8b): a callback that re-enters the emitter during Close no longer deadlocks (also fixes EmitWithCallback); domain emitters require ChipIngressLogger; ForDomain errors wrap ErrDomainEmittersDisabled / ErrUnknownDomain; the HTTP client rejects the switch; the chip client is closed if construction fails; more tests; the DomainEmitter contract is documented.
  • Rename (2cd0235): ChipIngressDomainEmittersEnabled → ChipDomainEmittersEnabled.
  • Benchmark alert: noise. BenchmarkKeystore_Sign/{nop,hex}/in-process doesn't touch beholder, and local runs at f49aea8 and 9b4d2d3 are the same (nop ~280 ns/op, hex ~302 ns/op).

thomaska
thomaska previously approved these changes Nov 4, 2026

@thomaska thomaska left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 💯 sgtm!

@4of9
4of9 marked this pull request as ready for review November 4, 2026 23:17
@4of9
4of9 requested review from a team as code owners November 4, 2026 23:17
Copilot AI balanced review requested due to automatic review settings November 4, 2026 23:17
@4of9
4of9 requested review from jmank88 and pavel-raykov November 4, 2026 23:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Domain validation and construction failures leave both initialized log providers running, leaking batch processors and the shared exporter connection without a returned Client to close them.

1 open issue
What changed in this PR

Adds opt-in, per-domain chip ingress emitters to Beholder, supporting telemetry routing to dedicated product-domain deployments.

Changes:

  • Adds ForDomain and EmitEvent with domain validation, dedicated buffers and connections, and no-ack publishing.
  • Integrates emitter lifecycle management, rate-limits failure logs, and moves queue-failure callbacks outside the service lock.
  • Updates the chip ingress dependency and adds routing, shutdown, and failure-path tests.
File Description
pkg/​beholder/​testdata/​config-example.json Shows domain emitters disabled by default.
pkg/​beholder/​httpclient.go Rejects domain emitters for HTTP clients.
pkg/​beholder/​error_log_limiter.go Adds per-error log rate limiting.
pkg/​beholder/​error_log_limiter_test.go Tests suppression counts and logging intervals.
pkg/​beholder/​domain_emitter.go Defines domain APIs and constructs dedicated emitters.
pkg/​beholder/​domain_emitter_test.go Tests routing, validation, and lifecycle behavior.
pkg/​beholder/​domain_emitter_failure_test.go Tests outages, buffering, concurrency, and cleanup.
pkg/​beholder/​config.go Adds the domain-emitter configuration flag.
pkg/​beholder/​client.go Integrates domain construction, lookup storage, and shutdown.
pkg/​beholder/​batch_emitter_service.go Adds CloudEvent emission, domain guards, and callback handling.
pkg/​beholder/​batch_emitter_service_reentrant_test.go Tests callback re-entry during shutdown.
go.sum Updates chip ingress dependency checksums.
go.mod Bumps chip ingress for no-ack batch support.

Review effort: Balanced

Comment thread pkg/beholder/client.go Outdated
if lggr == nil {
lggr = pkglogger.Nop()
}
chipIngressEnabled := cfg.ChipIngressEmitterEnabled || cfg.ChipIngressEmitterGRPCEndpoint != ""
Comment thread pkg/beholder/domain_emitter.go Outdated
func (*domainEmitter) Close() error { return nil }

// closeOnce wraps a service so that only its first Close does anything; later calls return nil.
type closeOnce struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historically we have tried to avoid these because they suppress problems that should be resolved, like unclear ownership. What was the purpose in this case? Is there a logical race to close from multiple places?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a race exactly: the double close request is deterministic, and it exists because of the ordering issue discussed earlier (#discussion_r4456767601, fixed in 9b4d2d3).

The engine runs the parent Close hook (which shuts down the OTel providers) before sub-service teardown. The domain emitters must drain before the providers die, or their drain-time metrics are lost, but they also need to stay sub-services for start rollback and health. So the same service is closed twice on purpose: Client.close drains it early, and the engine's later close is then a no-op. That's what closeOnce encodes.

Is that a bad idea? Should we do it differently?

@jmank88 jmank88 Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Being a sub-service usually means that the parent service owns the lifecycle. Do both operations need to strictly satisfy the Closer interface? Or could we call it something like stop or drain for the other case?

If we are stuck with Close in both cases, and if the engine always calls it first, how about wrapping only in the subservice case, and using something that suppresses close entirely? e.g. type noopCloser struct

@thomaska thomaska Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in #e141f09 if you could have a look please
In the rollback scenario though, the subservice close is never run, which in this case is ok as the domain emitters cannot fail start up, but in principle they could leak resources. This is ok right?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants